Project Management 


A Case Study on the Use of Quantitative 
Measures Applied to Basic Research 


Workshop on Performance Metrics 
for R&D Organizations 


Caroline S. Wagner 
March 2, 1999 


RAND 


Methodology 


- Identify government-sponsored international 
cooperative activities ($) 


+ Identify government-to-government 
agreements to support cooperation 


- Identify national goals for the ICST in 
earthquakes 


+ Match goals to measures 
Choose measures and conduct assessment 
Assess ‘lessons learned’ about assessment 
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A Framework for Applying Tools 


+ The framework focuses activities in the “second 
tier” of expected benefits construct 


« -in a matrix consisting of: 

A. reasons tor cooperation 
very large scale equipmert 
giobal nalure of subject 
unique foreign expertise or resource 
government mission 

8. type of cooperation 
collaborative research 
technical support 
operational support 


standards/database development 
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Three Measures Chosen 


Citation counts survey, 1985 and 1995 
Used Science Citation Index 


Standards-setting survey 
Surveyed foreign, US contacts 


Survey of US participants for foreign 
contribution 
Surveyed 45 U.S. grantees, one-quarter of talal 


Jointly Authored Papers Increased 


- Joint citations for earth sciences research rose 
to 351 in 1985 from 259 in 1995 


+ More joint papers were multinational in 
authorship in 1995 than in 1985 


. Japanese researchers were the most frequently 
cited co-authors, followed by Russia and China 


The U.S. Is Leveraging Dollars 


- Foreign contribution equats U.S. contribution 


Add in NASA and the US. is significantly leveraging 
foreign research dallars 


+ Three-fourths of surveyed projects had some 
foreign financial or in-kind contribution 
in 47% of projects, foreign contribution exceeded U.S. 
in 35% foreign contribution equaled U.S. 
- In 28% foreign contribution less than U.S. 
- Highest leverage was with Japan (NSF and 
NASA) 
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Case Study Suggested Parity, Benefit 


Case study on earthquake sciences and 
seismology suggested three things: 


Foreign partners contributed equally to joint 
Projects 

- Notin every case, but on average 

- In-kind, financial, or through access 


- Jointly authored articles are increasing 


- US is setting the standard in 80 percent of 
cases 


Quantitative hnding: Cooperation may be raising the ievel of 
excellence of science in the U.S. and else where 
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Assessing Science: It Can Be Done! 


» Tools need to be improved, true 

« Use what we have and improve them 

- Identify goals, benefits 

- Ask researchers how they measure themselves 


« Start with bibliometrics, expert judgment, 
survey research 


. Piace numbers in context 


+ Match a ‘numbers’ component with a ‘quality’ 
component 


RAND 


Nature of Benefit Depends on Mission 
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